Skip to content

feat(vortex-bench): wire SpatialBench into the bench orchestrator#8607

Open
HarukiMoriarty wants to merge 1 commit into
nemo/spatial-wkbfrom
nemo/spatial-wire-vx-bench
Open

feat(vortex-bench): wire SpatialBench into the bench orchestrator#8607
HarukiMoriarty wants to merge 1 commit into
nemo/spatial-wkbfrom
nemo/spatial-wire-vx-bench

Conversation

@HarukiMoriarty

@HarukiMoriarty HarukiMoriarty commented Jun 26, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Summary

Wires SpatialBench into the vx-bench / bench-orchestrator pipeline so it can be run end-to-end like the other benchmarks (datagen → Parquet → Vortex conversion → query). It builds on the WKB datagen landed in #8598.

Running command:

uv run --project bench-orchestrator vx-bench run spatialbench --engine duckdb --format parquet,vortex --opt scale-factor=N --queries 1,2,3,4,5,6,7,8,9 --iterations 3

Limitation

  • DuckDB-only. For now SpatialBench queries use DuckDB-specific ST_* spatial SQL that DataFusion has no functions for yet. There is a a single ad-hoc entry in BENCHMARK_ENGINES = { SPATIALBENCH: {DUCKDB} }.

  • No dictionary encoding / compaction on the WKB column. WKB geometry blobs are large and effectively unique, so running the dictionary builder over them balloons memory (tens of GB) for zero compression gain. The normal compaction path is preserved for every other column on every other benchmark.

  • Queries 10, 11, 12 is timeout simply because DuckDB poorly support on Spatial index.

Performance

SF=1.0

┏━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┓
┃ Query ┃ duckdb:parquet (base) ┃   duckdb:vortex ┃
┡━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━┩
│ 1     │                39.1ms │  14.6ms (0.37x) │
│ 2     │                72.1ms │  24.9ms (0.35x) │
│ 3     │                57.7ms │  20.1ms (0.35x) │
│ 4     │               113.1ms │  70.6ms (0.62x) │
│ 5     │               354.2ms │ 288.4ms (0.81x) │
│ 6     │               169.5ms │  91.6ms (0.54x) │
│ 7     │               156.7ms │  71.5ms (0.46x) │
│ 8     │               196.5ms │  80.3ms (0.41x) │
│ 9     │                20.3ms │  18.7ms (0.92x) │
└───────┴───────────────────────┴─────────────────┘

SF=10

┏━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┓
┃ Query ┃ duckdb:parquet (base) ┃   duckdb:vortex ┃
┡━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━┩
│ 1     │               158.6ms │ 114.4ms (0.72x) │
│ 2     │               255.1ms │ 219.3ms (0.86x) │
│ 3     │               229.2ms │ 181.5ms (0.79x) │
│ 4     │               184.5ms │ 134.3ms (0.73x) │
│ 5     │                 3.30s │   3.08s (0.93x) │
│ 6     │               476.4ms │ 348.9ms (0.73x) │
│ 7     │               918.2ms │ 961.2ms (1.05x) │
│ 8     │               980.6ms │ 926.7ms (0.94x) │
│ 9     │                33.7ms │  33.6ms (1.00x) │
└───────┴───────────────────────┴─────────────────┘

@HarukiMoriarty HarukiMoriarty added the changelog/feature A new feature label Jun 26, 2026
@HarukiMoriarty HarukiMoriarty requested a review from myrrc June 26, 2026 17:52
@HarukiMoriarty HarukiMoriarty mentioned this pull request Jun 26, 2026
Open
@HarukiMoriarty
feat: wire in the vx-benchmark
fdb0872 
Signed-off-by: Nemo Yu <zyu379@wisc.edu>
@HarukiMoriarty HarukiMoriarty force-pushed the nemo/spatial-wire-vx-bench branch from a0218b2 to fdb0872 Compare June 26, 2026 17:55
@codspeed-hq

codspeed-hq Bot commented Jun 26, 2026 •
edited
Loading

Copy link
Copy Markdown

Merging this PR will not alter performance

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

⚡ 4 improved benchmarks
❌ 2 regressed benchmarks
✅ 1583 untouched benchmarks
⏩ 4 skipped benchmarks1

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Mode Benchmark BASE HEAD Efficiency
❌ Simulation chunked_varbinview_into_canonical[(1000, 10)] 168.9 µs 205.5 µs -17.85%
❌ Simulation bitwise_not_vortex_buffer_mut[128] 215.3 ns 244.4 ns -11.93%
⚡ Simulation chunked_bool_canonical_into[(1000, 10)] 26.9 µs 16.7 µs +61.21%
⚡ Simulation chunked_varbinview_canonical_into[(100, 100)] 259.6 µs 224.5 µs +15.63%
⚡ Simulation chunked_varbinview_into_canonical[(100, 100)] 306.4 µs 271.1 µs +13.01%
⚡ Simulation eq_i64_constant 318.3 µs 288.6 µs +10.29%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.

Comparing nemo/spatial-wire-vx-bench (fdb0872) with nemo/spatial-wkb (c1635cb)

Open in CodSpeed

Footnotes

  1. 4 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@myrrc myrrc Awaiting requested review from myrrc

Assignees

No one assigned

Labels

changelog/feature A new feature

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@HarukiMoriarty